今天意外的发现clickhouse集群，clickhouse的日志路径一直打error日志。为了彻底把error消除，折腾了一番。

报错内容：

2021.11.30 23:49:41.774027 [ 44 ] {} void DB::StorageKafka::threadFunc(): Code: 73, e.displayText() = DB::Exception: Unknown format JSONAsString, Stack trace: 0.0x3512b60 StackTrace::StackTrace() /usr/bin/clickhouse 1.0x351cdaf DB::Exception::Exception(std::string const&, int) /usr/bin/clickhouse 2.0x657b9ee DB::FormatFactory::getCreators(std::string const&) const /usr/bin/clickhouse 3.0x657c85b DB::FormatFactory::getInput(std::string const&, DB::ReadBuffer&, DB::Block const&, DB::Context const&, unsigned long, std::function) const /usr/bin/clickhouse 4.0x69da7a3 DB::KafkaBlockInputStream::readImpl() /usr/bin/clickhouse 5.0x6073008 DB::IBlockInputStream::read() /usr/bin/clickhouse 6.0x61e8861 DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler::thread(std::shared_ptrDB::ThreadGroupStatus, unsigned long) /usr/bin/clickhouse 7.0x61e91eb ThreadFromGlobalPool::ThreadFromGlobalPool(void (DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler::&&)(std::shared_ptrDB::ThreadGroupStatus, unsigned long), DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler&&, std::shared_ptrDB::ThreadGroupStatus&&, unsigned long&)::{lambda()#1}::operator()() const /usr/bin/clickhouse 8.0x3554f13 ThreadPoolImplstd::thread::worker(std::_List_iteratorstd::thread) /usr/bin/clickhouse 9.0x791d69f ? /usr/bin/clickhouse 10.0x7fd3412c0e65 start_thread /usr/lib64/libpthread-2.17.so 11.0x7fd340de588d __clone /usr/lib64/libc-2.17.so (version 19.17.4.11)

Code: 73, e.displayText() = DB::Exception: Unknown format JSONAsString 这里是json解析出现问题。 KafkaBlockInputStream 这里跟kafka相关，所以初步猜测可能是kafka引擎解析字符串出现问题。

备注：system.tables表中的engine字段表示表引擎，

说明，这里查的时候疏忽了一点，error日志其实只出现了A节点，一直都没发现这个点。查system.tables时跑去了B节点。而恰巧，这张kafka引擎表，只建在了A节点，是个本地表。所以才一直没在system.table中找到kafka引擎相关的表。【这一切都是后面才恍然大悟的】

2021.12.01 00:22:24.712135 [ 26 ] {} StorageKafka (kafka_user_behavior_src_test): Execution took 501 ms. 2021.12.01 00:22:25.212277 [ 43 ] {} StorageKafka (kafka_user_behavior_src_test): Started streaming to 1 attached views 2021.12.01 00:22:25.212868 [ 70 ] {} StorageKafka (kafka_user_behavior_src_test): Already subscribed to topics: 2021.12.01 00:22:25.212873 [ 52 ] {} StorageKafka (kafka_user_behavior_src_test): Already subscribed to topics: 2021.12.01 00:22:25.212962 [ 70 ] {} StorageKafka (kafka_user_behavior_src_test): Already assigned to topics: 2021.12.01 00:22:25.212998 [ 52 ] {} StorageKafka (kafka_user_behavior_src_test): Already assigned to topics: 2021.12.01 00:22:25.713303 [ 70 ] {} StorageKafka (kafka_user_behavior_src_test): Stalled 2021.12.01 00:22:25.713311 [ 52 ] {} StorageKafka (kafka_user_behavior_src_test): Stalled 2021.12.01 00:22:25.713694 [ 43 ] {} UnionBlockInputStream: Waiting for threads to finish 2021.12.01 00:22:25.713729 [ 43 ] {} UnionBlockInputStream: Waited for threads to finish 2021.12.01 00:22:25.713810 [ 43 ] {} virtual DB::UnionBlockInputStream::~UnionBlockInputStream(): Code: 73, e.displayText() = DB::Exception: Unknown format JSONAsString, Stack trace: 0.0x3512b60 StackTrace::StackTrace() /usr/bin/clickhouse 1.0x351cdaf DB::Exception::Exception(std::string const&, int) /usr/bin/clickhouse 2 0x657b9ee DB::FormatFactory::getCreators(std::string const&) const /usr/bin/clickhouse 3 0x657c85b DB::FormatFactory::getInput(std::string const&, DB::ReadBuffer&, DB::Block const&, DB::Context const&, unsigned long, std::function) const /usr/bin/clickhouse 4.0x69da7a3 DB::KafkaBlockInputStream::readImpl() /usr/bin/clickhouse 5.0x6073008 DB::IBlockInputStream::read() /usr/bin/clickhouse 6.0x61e8861 DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler::thread(std::shared_ptrDB::ThreadGroupStatus, unsigned long) /usr/bin/clickhouse 7.0x61e91eb ThreadFromGlobalPool::ThreadFromGlobalPool(void (DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler::&&)(std::shared_ptrDB::ThreadGroupStatus, unsigned long), DB::ParallelInputsProcessorDB::UnionBlockInputStream::Handler&&, std::shared_ptrDB::ThreadGroupStatus&&, unsigned long&)::{lambda()#1}::operator()() const /usr/bin/clickhouse 8.0x3554f13 ThreadPoolImplstd::thread::worker(std::_List_iteratorstd::thread) /usr/bin/clickhouse 9.0x791d69f ? /usr/bin/clickhouse 10.0x7f4154b86e65 start_thread /usr/lib64/libpthread-2.17.so 11.0x7f41546ab88d __clone /usr/lib64/libc-2.17.so (version 19.17.4.11)

6 . 再后来直到发现节点B没有error日志，这才意识到，会不会是该表只建了本地表，去A节点的clickhouse中的system.tables一查，还真是如此。第五点的疑问也迎刃而解了，kafka_user_behavior_src_test其实并非groupid，而是KafkaEngine表的表名。在节点A就能找到这张表。

其实这里在此之前，还尝试去各kafka集群里搜groups，不过均没搜到。方向都偏了。

如果日志满足不了您分析定位问题，不妨修改一下clickhouse的配置文件，将error级别修改为trace，再配合观察info日志与error日志，可以帮您将问题再缩小一个范围。真实有用！！！