Android 录制手机内部声音（screen recorder）framework层问题分析 |
您所在的位置：网站首页 › 手机录屏录制内部声音小 › Android 录制手机内部声音（screen recorder）framework层问题分析 |
Android 录制手机内部声音（screen recorder）framework层问题分析
|
目录 背景： 正文： 1. 先播放声音，再打开录屏工具，是如果保证正常录制的？ 2. 对于secondary output，他的输出设备是怎么选择的？ 3. 数据流是如何运转的？ 背景：Android的screen recorder应用层中对于手机内部声音录制部分的实现是通过AudioPlaybackCaptureConfiguration来进行录制系统声音，实现如下： 连接：ScreenInternalAudioRecorder.java - Android Code Search 88 int size = AudioRecord.getMinBufferSize( 89 mConfig.sampleRate, mConfig.channelInMask, 90 mConfig.encoding) * 2; 91 92 Log.d(TAG, "audio buffer size: " + size); 93 94 AudioFormat format = new AudioFormat.Builder() 95 .setEncoding(mConfig.encoding) 96 .setSampleRate(mConfig.sampleRate) 97 .setChannelMask(mConfig.channelOutMask) 98 .build(); 99 100 AudioPlaybackCaptureConfiguration playbackConfig = 101 new AudioPlaybackCaptureConfiguration.Builder(mMediaProjection) 102 .addMatchingUsage(AudioAttributes.USAGE_MEDIA) 103 .addMatchingUsage(AudioAttributes.USAGE_UNKNOWN) 104 .addMatchingUsage(AudioAttributes.USAGE_GAME) 105 .build(); 106 // 由于有AudioPlaybackCaptureConfiguration的存在，所以会触发audiopolicy的注册， 并且通过policy获取到record的行为，这个recorder的source为REMOTE_SUBMIX。 具体在audiorecord::buildAudioPlaybackCaptureRecord中 107 mAudioRecord = new AudioRecord.Builder() 108 .setAudioFormat(format) 109 .setAudioPlaybackCaptureConfig(playbackConfig) 110 .build();应用层面以及policymix这块这篇文章里面不打算细讲，主要是针对上面几个疑问进行解释。 正文： 1. 先播放声音，再打开录屏工具，是如果保证正常录制的？不管后台有没有声音在播放，我们的入口都是AudioRecord: 在AudioRecord创建的时候，会通过AudioFlinger::createRecord()创建record handle，在创建record handle之前会先从AudioPolicyManager中获取到对应的input handle的，这个input handle是根据AudioRecord中的属性，从AudioPolicyManager中获取。audio record中比较重要的属性如source（REMOTE SUBMIX）以及audiomananger中policyMix等信息来创建、获取对应的input handle。 另外一点是在AudioRecord::start的时候，会触发AudioPolicyManager的startInput的逻辑，startInput中对REMOTE SUBMIX的判断，从而往音频系统里面添加AUDIO_DEVICE_OUT_REMOTE_SUBMIX的设备，如下面代码： // automatically enable the remote submix output when input is started if not // used by a policy mix of type MIX_TYPE_RECORDERS // For remote submix (a virtual device), we open only one input per capture request. if (audio_is_remote_submix_device(inputDesc->getDeviceType())) { String8 address = String8(""); if (policyMix == nullptr) { address = String8("0"); } else if (policyMix->mMixType == MIX_TYPE_PLAYERS) { address = policyMix->mDeviceAddress; } if (address != "") { setDeviceConnectionStateInt(AUDIO_DEVICE_OUT_REMOTE_SUBMIX, AUDIO_POLICY_DEVICE_STATE_AVAILABLE, address, "remote-submix", AUDIO_FORMAT_DEFAULT); } }下面这个是policymix在AudioPolicyManager中的注册信息： Audio Policy Mix: Audio Policy Mix 1 (0xb400007de1102a80): - mix type: MIX_TYPE_PLAYERS - Route Flags: MIX_ROUTE_FLAG_RENDER|MIX_ROUTE_FLAG_LOOP_BACK|MIX_ROUTE_FLAG_LOOP_BACK_AND_RENDER|MIX_ROUTE_FLAG_ALL - device type: AUDIO_DEVICE_OUT_REMOTE_SUBMIX - device address: -785045321:ap:1mixp:0 - output: 69 - Criterion 0: RULE_MATCH_ATTRIBUTE_USAGE AUDIO_USAGE_MEDIA - Criterion 1: RULE_MATCH_ATTRIBUTE_USAGE AUDIO_USAGE_VOICE_COMMUNICATION接下来是重点：下面是调用的流程 AudioRecord::start-》startInput()-》setDeviceConnectionStateInt-》checkForDeviceAndOutputChanges-》checkSecondaryOutputs-》invalidateStream()->AudioTrack::restoreTrack_l()->AudioFlinger::createTrack()->TrackHandle::start()->PatchTrack::start() 下面是AudioPolicyManager中重要函数checkSecondaryOutputs的代码解析： 总结：遍历所有的outputs的thread，查找所有线程上的clients是否有符合policymix的规则的，然后在确认下有无该policymix的secondaryOutput存在，如果存在，则后续对该client所在播放的stream type进行invalidate。 void AudioPolicyManager::checkSecondaryOutputs() { std::set streamsToInvalidate; // 遍历所有的播放线程 for (size_t i = 0; i < mOutputs.size(); i++) { const sp& outputDescriptor = mOutputs[i]; // 看看这些播放线程里面的所有播放的client端 for (const sp& client : outputDescriptor->getClientIterable()) { sp primaryMix; std::vector secondaryMixes; // 这里会判断这个client是否允许你，以及是否符合你的policymix中的条件， // 如果是的话，则会往secondaryMixes中追加该mix， // 表明这个这个client播放的声音可以采用该policy mix status_t status = mPolicyMixes.getOutputForAttr(client->attributes(), client->uid(), client->flags(), primaryMix, &secondaryMixes); std::vector secondaryDescs; // 遍历所有的secondary mix，对于有output信息的mix存放到secondaryDescs列表中。 // 在checkSecondaryOutputs之前的checkOutputsForDevice的时候， // 会去打开REMOTE_SUBMIX的output，而将secondary mix中setOutput指向 // 新打开的这个handle for (auto &secondaryMix : secondaryMixes) { sp outputDesc = secondaryMix->getOutput(); if (outputDesc != nullptr && outputDesc->mIoHandle != AUDIO_IO_HANDLE_NONE) { secondaryDescs.push_back(outputDesc); } } if (status != OK || !std::equal(client->getSecondaryOutputs().begin(), client->getSecondaryOutputs().end(), secondaryDescs.begin(), secondaryDescs.end())) { streamsToInvalidate.insert(client->stream()); } } } for (audio_stream_type_t stream : streamsToInvalidate) { // 将会触发如music类型的track进入invalidate状态，从而recreate的逻辑 mpClientInterface->invalidateStream(stream); } }下面是AudioFlinger::createTrack()中针对secondaryOutput的部分代码： 总结：根据是否有secondaryOutput来创建对应的patchRecord和pathTrack，并将patchTrack添加到secondaryThread中，patchTrack跟patchRecord也会彼此建立Peer Proxy，还需要添加到track中。 这块应该是数据流的重点，后面应该会再整理下： if (lStatus == NO_ERROR) { // Connect secondary outputs. Failure on a secondary output must not imped the primary // Any secondary output setup failure will lead to a desync between the AP and AF until // the track is destroyed. TeePatches teePatches; for (audio_io_handle_t secondaryOutput : secondaryOutputs) { PlaybackThread *secondaryThread = checkPlaybackThread_l(secondaryOutput); if (secondaryThread == NULL) { ALOGE("no playback thread found for secondary output %d", output.outputId); continue; } size_t sourceFrameCount = thread->frameCount() * output.sampleRate / thread->sampleRate(); size_t sinkFrameCount = secondaryThread->frameCount() * output.sampleRate / secondaryThread->sampleRate(); // If the secondary output has just been opened, the first secondaryThread write // will not block as it will fill the empty startup buffer of the HAL, // so a second sink buffer needs to be ready for the immediate next blocking write. // Additionally, have a margin of one main thread buffer as the scheduling jitter // can reorder the writes (eg if thread A&B have the same write intervale, // the scheduler could schedule AB...BA) size_t frameCountToBeReady = 2 * sinkFrameCount + sourceFrameCount; // Total secondary output buffer must be at least as the read frames plus // the margin of a few buffers on both sides in case the // threads scheduling has some jitter. // That value should not impact latency as the secondary track is started before // its buffer is full, see frameCountToBeReady. size_t frameCount = frameCountToBeReady + 2 * (sourceFrameCount + sinkFrameCount); // The frameCount should also not be smaller than the secondary thread min frame // count size_t minFrameCount = AudioSystem::calculateMinFrameCount( [&] { Mutex::Autolock _l(secondaryThread->mLock); return secondaryThread->latency_l(); }(), secondaryThread->mNormalFrameCount, secondaryThread->mSampleRate, output.sampleRate, input.speed); frameCount = std::max(frameCount, minFrameCount); using namespace std::chrono_literals; auto inChannelMask = audio_channel_mask_out_to_in(input.config.channel_mask); sp patchRecord = new RecordThread::PatchRecord(nullptr /* thread */, output.sampleRate, inChannelMask, input.config.format, frameCount, NULL /* buffer */, (size_t)0 /* bufferSize */, AUDIO_INPUT_FLAG_DIRECT, 0ns /* timeout */); status_t status = patchRecord->initCheck(); if (status != NO_ERROR) { ALOGE("Secondary output patchRecord init failed: %d", status); continue; } // TODO: We could check compatibility of the secondaryThread with the PatchTrack // for fast usage: thread has fast mixer, sample rate matches, etc.; // for now, we exclude fast tracks by removing the Fast flag. const audio_output_flags_t outputFlags = (audio_output_flags_t)(output.flags & ~AUDIO_OUTPUT_FLAG_FAST); sp patchTrack = new PlaybackThread::PatchTrack(secondaryThread, streamType, output.sampleRate, input.config.channel_mask, input.config.format, frameCount, patchRecord->buffer(), patchRecord->bufferSize(), outputFlags, 0ns /* timeout */, frameCountToBeReady); status = patchTrack->initCheck(); if (status != NO_ERROR) { ALOGE("Secondary output patchTrack init failed: %d", status); continue; } teePatches.push_back({patchRecord, patchTrack}); secondaryThread->addPatchTrack(patchTrack); // In case the downstream patchTrack on the secondaryThread temporarily outlives // our created track, ensure the corresponding patchRecord is still alive. patchTrack->setPeerProxy(patchRecord, true /* holdReference */); patchRecord->setPeerProxy(patchTrack, false /* holdReference */); } track->setTeePatches(std::move(teePatches)); } 2. 对于secondary output，他的输出设备是怎么选择的？在上面的部分中有提到在checkSecondaryOutputs之前，会进行checkOutputsForDevice的操作，该部分的操作将会创建OUT REMOTE SUBMIX的output，在创建结束的时候，会进行首次的setOutputDevice的行为： AudioPolicyManager::checkOutputsForDevice() desc = new SwAudioOutputDescriptor(profile, mpClientInterface); audio_io_handle_t output = AUDIO_IO_HANDLE_NONE; status_t status = desc->open(nullptr, DeviceVector(device), AUDIO_STREAM_DEFAULT, AUDIO_OUTPUT_FLAG_NONE, &output); if (status == NO_ERROR) { if (output != AUDIO_IO_HANDLE_NONE) { addOutput(output, desc); if (audio_is_remote_submix_device(deviceType) && address != "0") { sp policyMix; if (mPolicyMixes.getAudioPolicyMix(deviceType, address, policyMix) == NO_ERROR) { policyMix->setOutput(desc); desc->mPolicyMix = policyMix; } } } } else { output = AUDIO_IO_HANDLE_NONE; } if (output == AUDIO_IO_HANDLE_NONE) { } else { outputs.add(output); if (device_distinguishes_on_address(deviceType)) { setOutputDevices(desc, DeviceVector(device), true/*force*/, 0/*delay*/, NULL/*patch handle*/); } } } 3. 数据流是如何运转的？Track::releaseBuffer()->interceptBuffer()->patchRecord::writeFrames() 将数据写到bufferProvider -> A. patchRecord::getNextBuffer(BufferProvider) -> A.1. pathTrack::obtainBuffer(ProxyBuffer)->mProxy::obtainBuffer() 获取大小给record参考 A.2. RecordTrack::getNextBuffer(ProxyBuffer)->mServerProxy::obtainBuffer() bufferprovider的指针指向record的cblk的空间 B. memcpy 将数据拷贝到获取到的BufferProvider，也就是拷贝到record的cblk下的空间，而patchTrack的mBuffer其实是指向patchRecord的buffer的，这个在audioflinger中创建PatchTrack的时候已经传递过来了 C. patchRecord::releaseBuffer(BufferProvider) -> C.1. patchTrack::releaseBuffer(ProxyBuffer) ->proxy::releaseBuffer(ProxyBuffer) C.2. TrackBase::releaseBuffer() -> pathRecord::serverProxy::releaseBuffer(ServerProxyBuffer) (上面这部分的pathRecord正符合上行的逻辑，server端填充数据，client端使用数据，releasebuffer，而server端填充数据的多少是根据patchTrack中client端去获取的buffer的大小， 同样patchTrack也完成了客户端填写数据的逻辑，填充buffer，releaseBuffer，等待audiomixer来消费数据，并更新buffer的信息)
PatchTrack在output线程中，一直处在如果数据充足，那就往底层写数据的逻辑，而PatchRecord，则是在等AudioTrack被mixer releaseBuffer的时机，来将数据写到obtainBuffer中buffer中。而这个buffer又是PatchTrack的buffer，这样patchTrack只需要通过proxy来进行releaseBuffer的操作，从而更新mCblk里面的信息。这样PatchTrack所在的output线程就可以捞到数据，往底层写了。 所以根据以上，可以判断出来数据流是从AudioTrack送到PatchRecord，再送到PatchTrack送到remote submix的hal的，然后app则通过自己的AudioRecord将数据给捞上来。 所以重点就是patchRecord跟patchTrack都是Track这端传输数据的产物，以及他们共用buffer的这两点。 |
【本文地址】
今日新闻 |
推荐新闻 |