zookeeper各种报错、原因及解决方法汇总（持续更新） |
您所在的位置：网站首页 › jline是谁 › zookeeper各种报错、原因及解决方法汇总（持续更新） |
|
[root@iZ23nn1p4mjZ zookeeper-3.4.10]# bin/zkCli.sh -server localhost:2181,localhost:2182,localhost:2183
Connecting to localhost:2181,localhost:2182,localhost:2183
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181,localhost:2182,localhost:2183(CONNECTING) 0] create -s /[sPacktPub]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:706)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:371)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:331)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
[root@iZ23nn1p4mjZ zookeeper-3.4.10]# ls /
上述问题近发生在单机三节点模式时。 ZooKeeper自定义数据日志目录ZooKeeper在启动时会将zookeeper.out输出到当前目录，不仅不友好，有时候可能会因为目录权限问题引发一些不必要的麻烦。 解决方案：设置ZOO_LOG_DIR环境变量，也可以改在shell中写，如下： if [ $ZOO_LOG_DIR ] ; then export ZOO_LOG_DIR=/usr/local/zookeeper/logs fi if [ -z $ZOO_LOG_DIR ];then export ZOO_LOG_DIR=/usr/local/zookeeper/logs ficonf/log4j.properties中定义： zookeeper.log.dir=/usr/local/zookeeper/logs zookeeper.tracelog.dir=/usr/local/zookeeper/logs有时候在网络变化、抖动或者服务器负载高的情况下会出现下列异常： 14:18:06.812 [main] ERROR org.apache.curator.ConnectionState - Connection timed out for connection string (127.0.0.1:2181) and timeout (5000) / elapsed (59220) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss重启应用后仍然如此，重启zk后便自动恢复。 最近有个奇怪的现象{20200420更新}，只要连接非本地，客户端IDEA中总是第一次连接的时候超时，第二次就成功了，每个客户端均如此。如下： 19:51:53.245 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=xx.xx.xx.xx:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@310d05c5 19:51:58.334 [main] ERROR org.apache.curator.ConnectionState - Connection timed out for connection string (xx.xx.xx.xx:2181) and timeout (5000) / elapsed (5104) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225) [curator-client-2.13.0.jar:?] at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94) [curator-client-2.13.0.jar:?] at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117) [curator-client-2.13.0.jar:?] at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723) [curator-framework-2.13.0.jar:?] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) [curator-client-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454) [curator-framework-2.13.0.jar:?] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) [curator-framework-2.13.0.jar:?] at com.alibaba.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.createPersistent(CuratorZookeeperClient.java:96) [classes/:1.0.17]连接超时时间太短，加大即可，有需求可有偿解决。
curator KeeperErrorCode = Unimplemented异常，如下： org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /benchmark/test-key-1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:103) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1525) at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1189) at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1166) at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)ZooKeeper 3.5.x Curator 4.0 has a hard dependency on ZooKeeper 3.5.x If you are using ZooKeeper 3.5.x there's nothing additional to do - just use Curator 4.0 ZooKeeper 3.4.x Curator 4.0 supports ZooKeeper 3.4.x ensembles in a soft-compatibility mode. To use this mode you must exclude ZooKeeper when adding Curator to your dependency management tool. Curator 4.0默认不支持zk 3.4，必须兼容性处理，如下： org.apache.zookeeper zookeeper 3.4.12 org.apache.curator curator-recipes 4.0.0 org.apache.zookeeper zookeeper见http://curator.apache.org/zk-compatibility.html 其他zk自身及zk与dubbo相关的异常可参见：https://www.cnblogs.com/zhjh256/p/11679639.html 2019-11-04 12:05:17,165 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data from client sessionid 0x1001e6b86a30003, likely client has closed socket 2019-11-04 12:05:17,166 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /10.2.104.6:22804 which had sessionid 0x1001e6b86a30003 2019-11-04 12:05:17,240 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data from client sessionid 0x1001e6b86a30000, likely client has closed socket 2019-11-04 12:05:17,240 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /10.2.104.6:22665 which had sessionid 0x1001e6b86a30000 2019-11-04 12:05:17,241 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data from client sessionid 0x1001e6b86a30001, likely client has closed socket 2019-11-04 12:05:17,241 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /10.2.104.6:22672 which had sessionid 0x1001e6b86a30001 2019-11-04 12:05:17,241 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data from client sessionid 0x1001e6b86a30002, likely client has closed socket 2019-11-04 12:05:17,242 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /10.2.104.6:22673 which had sessionid 0x1001e6b86a30002 2019-11-04 12:05:47,455 [myid:] - INFO [SessionTracker:ZooKeeperServer@354] - Expiring session 0x1001e6b86a30002, timeout of 40000ms exceeded 2019-11-04 12:05:47,456 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1001e6b86a30002 2019-11-04 12:05:51,456 [myid:] - INFO [SessionTracker:ZooKeeperServer@354] - Expiring session 0x1001e6b86a30003, timeout of 40000ms exceeded 2019-11-04 12:05:51,456 [myid:] - INFO [SessionTracker:ZooKeeperServer@354] - Expiring session 0x1001e6b86a30000, timeout of 40000ms exceeded 2019-11-04 12:05:51,456 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1001e6b86a30003 2019-11-04 12:05:51,457 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1001e6b86a30000 2019-11-04 12:05:53,456 [myid:] - INFO [SessionTracker:ZooKeeperServer@354] - Expiring session 0x1001e6b86a30001, timeout of 40000ms exceeded 2019-11-04 12:05:53,457 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x1001e6b86a30001该异常看起来和https://www.cnblogs.com/zhyg/p/11089736.html所述很像。 zookeeper服务器日志报连接数超出： 2020-04-16 17:16:11,135 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,241 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,371 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,381 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,463 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,527 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,553 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,580 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,609 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,686 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,796 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,818 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60 2020-04-16 17:16:11,853 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /10.20.25.225 - max is 60该原因大概率是客户端有连接泄露，可通过netstat -ano | grep 2181分析，如下： [root@hs-10-20-37-72 logs]# lsof -i:2181 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 2706 root 24u IPv6 18738 0t0 TCP *:eforward (LISTEN) java 2706 root 25u IPv6 3786862 0t0 TCP 10.20.25.225:eforward->10.20.25.225:63464 (ESTABLISHED) java 2706 root 26u IPv6 4748980 0t0 TCP 10.20.25.225:eforward->10.20.25.225:59063 (ESTABLISHED) java 2706 root 27u IPv6 6781917 0t0 TCP 10.20.25.225:eforward->10.20.25.225:51577 (ESTABLISHED) java 2706 root 28u IPv6 6781920 0t0 TCP 10.20.25.225:eforward->10.20.25.225:51578 (ESTABLISHED) java 2706 root 30u IPv6 6784242 0t0 TCP 10.20.25.225:eforward->10.20.25.225:51880 (ESTABLISHED) java 2706 root 31u IPv6 6784604 0t0 TCP 10.20.25.225:eforward->10.20.25.225:51989 (ESTABLISHED) java 2706 root 32u IPv6 6784610 0t0 TCP 10.20.25.225:eforward->10.20.25.225:51990 (ESTABLISHED) java 2706 root 33u IPv6 6788885 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52069 (ESTABLISHED) java 2706 root 34u IPv6 4412727 0t0 TCP 10.20.25.225:eforward->10.20.25.225:12441 (ESTABLISHED) java 2706 root 35u IPv6 6789241 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52127 (ESTABLISHED) java 2706 root 36u IPv6 6789263 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52130 (ESTABLISHED) java 2706 root 37u IPv6 6789329 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52140 (ESTABLISHED) java 2706 root 38u IPv6 6789422 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52149 (ESTABLISHED) java 2706 root 39u IPv6 6789572 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52166 (ESTABLISHED) java 2706 root 40u IPv6 6789693 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52183 (ESTABLISHED) java 2706 root 41u IPv6 6789807 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52200 (ESTABLISHED) java 2706 root 42u IPv6 6789981 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52220 (ESTABLISHED) java 2706 root 43u IPv6 6790066 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52239 (ESTABLISHED) java 2706 root 44u IPv6 6790173 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52250 (ESTABLISHED) java 2706 root 45u IPv6 6790238 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52259 (ESTABLISHED) java 2706 root 46u IPv6 6790287 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52272 (ESTABLISHED) java 2706 root 47u IPv6 6790356 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52285 (ESTABLISHED) java 2706 root 48u IPv6 6790447 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52297 (ESTABLISHED) java 2706 root 49u IPv6 6790485 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52307 (ESTABLISHED) java 2706 root 50u IPv6 6790597 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52337 (ESTABLISHED) java 2706 root 51u IPv6 6790638 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52343 (ESTABLISHED) java 2706 root 52u IPv6 6790677 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52356 (ESTABLISHED) java 2706 root 53u IPv6 6790726 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52368 (ESTABLISHED) java 2706 root 54u IPv6 6790842 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52387 (ESTABLISHED) java 2706 root 55u IPv6 6790871 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52395 (ESTABLISHED) java 2706 root 56u IPv6 6790883 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52402 (ESTABLISHED) java 2706 root 57u IPv6 6790992 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52415 (ESTABLISHED) java 2706 root 58u IPv6 6791084 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52433 (ESTABLISHED) java 2706 root 59u IPv6 6791118 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52449 (ESTABLISHED) java 2706 root 60u IPv6 6791187 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52460 (ESTABLISHED) java 2706 root 61u IPv6 6791270 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52479 (ESTABLISHED) java 2706 root 62u IPv6 6791334 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52488 (ESTABLISHED) java 2706 root 63u IPv6 6791427 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52501 (ESTABLISHED) java 2706 root 64u IPv6 6791470 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52518 (ESTABLISHED) java 2706 root 65u IPv6 6791542 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52531 (ESTABLISHED) java 2706 root 66u IPv6 6791631 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52545 (ESTABLISHED) java 2706 root 67u IPv6 6791669 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52555 (ESTABLISHED) java 2706 root 68u IPv6 6791719 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52566 (ESTABLISHED) java 2706 root 69u IPv6 6791797 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52581 (ESTABLISHED) java 2706 root 70u IPv6 6791847 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52592 (ESTABLISHED) java 2706 root 71u IPv6 6791883 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52602 (ESTABLISHED) java 2706 root 72u IPv6 6791931 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52615 (ESTABLISHED) java 2706 root 73u IPv6 6792017 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52627 (ESTABLISHED) java 2706 root 74u IPv6 6792056 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52641 (ESTABLISHED) java 2706 root 75u IPv6 6792090 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52648 (ESTABLISHED) java 2706 root 76u IPv6 6792199 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52661 (ESTABLISHED) java 2706 root 77u IPv6 6792283 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52681 (ESTABLISHED) java 2706 root 78u IPv6 6792299 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52688 (ESTABLISHED) java 2706 root 79u IPv6 6792345 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52698 (ESTABLISHED) java 2706 root 80u IPv6 6792413 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52709 (ESTABLISHED) java 2706 root 81u IPv6 6792495 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52727 (ESTABLISHED) java 2706 root 82u IPv6 6792563 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52736 (ESTABLISHED) java 2706 root 83u IPv6 6792625 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52745 (ESTABLISHED) java 2706 root 84u IPv6 6792674 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52758 (ESTABLISHED) java 2706 root 85u IPv6 6792755 0t0 TCP 10.20.25.225:eforward->10.20.25.225:52775 (ESTABLISHED) java 2706 root 86u IPv6 6290356 0t0 TCP 10.20.25.225:eforward->192.168.166.145:34608 (ESTABLISHED) java 2871 root 176u IPv6 4748979 0t0 TCP 10.20.25.225:59063->10.20.25.225:eforward (ESTABLISHED)找到对应的进程和服务器。然后分析泄露，通常大概率是因为自己手写zk相关连接和监控、而不是使用客户端库如curator所致。 21:12:03.568 [pool-taskSchedule-thread-54-SendThread(10.20.25.225:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.20.25.225/10.20.25.225:2181. Will not attempt to authenticate using SASL (unknown error) 21:12:03.568 [pool-taskSchedule-thread-54-SendThread(10.20.25.225:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.20.25.225/10.20.25.225:2181, initiating session 21:12:03.576 [pool-taskSchedule-thread-54-SendThread(10.20.25.225:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server 10.20.25.225/10.20.25.225:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_191] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_191] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_191] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_191] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_191] at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) ~[zookeeper-3.4.8.jar!/:3.4.8--1] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) ~[zookeeper-3.4.8.jar!/:3.4.8--1] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) [zookeeper-3.4.8.jar!/:3.4.8--1] 21:12:03.729 [pool-taskSchedule-thread-52-SendThread(10.20.25.225:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.20.25.225/10.20.25.225:2181. Will not attempt to authenticate using SASL (unknown error) 21:12:03.730 [pool-taskSchedule-thread-52-SendThread(10.20.25.225:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.20.25.225/10.20.25.225:2181, initiating session 21:12:03.730 [pool-taskSchedule-thread-52-SendThread(10.20.25.225:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server 10.20.25.225/10.20.25.225:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_191] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_191] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_191] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_191] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_191] at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) ~[zookeeper-3.4.8.jar!/:3.4.8--1] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) ~[zookeeper-3.4.8.jar!/:3.4.8--1] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) [zookeeper-3.4.8.jar!/:3.4.8--1]Caused by: org.apache.zookeeper.KeeperException$OperationTimeoutException: KeeperErrorCode = OperationTimeout at com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter.init(ZookeeperRegistryCenter.java:102) ~[elastic-job-common-core-2.1.5.jar:?] 检查网络是否正常，ping/telnet看下，基本上就这一个原因。 执行[[email protected] bin]# echo conf | nc localhost 2181报下列输出This ZooKeeper instance is not currently serving requests 说明zookeeper还活着，但不提供服务了。此时要查看zk的日志，dataLogDir指定的目录，一般里面可以看到为什么zk不提供服务了。如下： 2020-08-17 00:13:20,050 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:20,252 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:20,654 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:21,463 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:23,065 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:26,271 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:32,679 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x103c00000494 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LEADING (my state) 2020-08-17 00:13:40,040 [myid:0] - WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:QuorumPeer@998] - Unexpected exception java.lang.InterruptedException: Timeout while waiting for epoch to be acked by quorum at org.apache.zookeeper.server.quorum.Leader.waitForEpochAck(Leader.java:936) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:415) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:995) 2020-08-17 00:13:40,040 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:Leader@520] - Shutting down 2020-08-17 00:13:40,040 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:Leader@526] - Shutdown called java.lang.Exception: shutdown Leader! reason: Forcing shutdown at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:526) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1001) 2020-08-17 00:13:40,041 [myid:0] - INFO [LearnerCnxAcceptor-/10.20.30.17:2182:Leader$LearnerCnxAcceptor@340] - exception while shutting down acceptor: java.net.SocketException: Socket closed 2020-08-17 00:13:40,041 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:QuorumPeer@910] - LOOKING 2020-08-17 00:13:40,042 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@813] - New election. My id = 0, proposed zxid=0x103c00000496 2020-08-17 00:13:40,042 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 0 (n.leader), 0x103c00000496 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 0 (n.sid), 0x103c (n.peerEpoch) LOOKING (my state) 2020-08-17 00:13:40,043 [myid:0] - WARN [WorkerSender[myid=0]:QuorumCnxManager@584] - Cannot open channel to 1 at election address/*原因和解决方法可见https://blog.csdn.net/everl_1/article/details/52253513*/ /10.20.30.13:2183 java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:534) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:454) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:435) at java.lang.Thread.run(Thread.java:748) 2020-08-17 00:13:40,043 [myid:0] - INFO [WorkerSender[myid=0]:QuorumPeer$QuorumServer@185] - Resolved hostname: 10.20.30.13 to address: /10.20.30.13 2020-08-17 00:13:40,045 [myid:0] - INFO [WorkerReceiver[myid=0]:FastLeaderElection@595] - Notification: 1 (message format version), 0 (n.leader), 0x103c00000496 (n.zxid), 0x17af2 (n.round), LOOKING (n.state), 2 (n.sid), 0x103c (n.peerEpoch) LOOKING (my state) 2020-08-17 00:13:40,245 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:QuorumPeer@992] - LEADING 2020-08-17 00:13:40,263 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@174] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datad ir /home/component/zookeeper/log/version-2 snapdir /home/component/zookeeper/data/version-2 2020-08-17 00:13:40,264 [myid:0] - INFO [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181:Leader@380] - LEADING - LEADER ELECTION TOOK - 222 2020-08-17 00:13:41,271 [myid:0] - INFO [LearnerHandler-/10.20.30.18:11030:LearnerHandler@346] - Follower sid: 2 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1efeac 2020-08-17 00:13:41,307 [myid:0] - ERROR [LearnerHandler-/10.20.30.18:11030:LearnerHandler@648] - Unexpected exception causing shutdown while sock still open java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:373) 2020-08-17 00:13:41,308 [myid:0] - WARN [LearnerHandler-/10.20.30.18:11030:LearnerHandler@661] - ******* GOODBYE /10.20.30.18:11030 ********所以，此处根本原因是集群的其他节点连不上，导致节点异常退出。但是socket没有关闭，这样是为了在其他节点恢复后，集群整体自动恢复。 解决Zookeeper报错:conf is not executed because it is not in the whitelist的解决办法[root@hs-10-20-37-72 ~]# echo wchs | nc 127.0.0.1 21812 connections watching 48 pathsTotal watches:48[root@hs-10-20-37-72 ~]# echo wchc | nc 127.0.0.1 2181wchc is not executed because it is not in the whitelist.[root@hs-10-20-37-72 ~]# echo wchp | nc 127.0.0.1 2181wchp is not executed because it is not in the whitelist. 解决方法：cd /usr/local/zookeeper/conf下，用vim打开zoo.cfg，在最后添加: 4lw.commands.whitelist=*重启zk。 ZooKeeper 支持某些特定的四字命令字母与其的交互。它们大多是查询命令，用来获取 ZooKeeper 服务的当前状态及相关信息。用户在客户端可以通过 telnet 或 nc 向 ZooKeeper 提交相应的命令。 ZooKeeper 常用四字命令见下表 1 所示： 表 1 ： ZooKeeper 四字命令 ZooKeeper 四字命令 功能描述 conf 输出相关服务配置的详细信息。 cons 列出所有连接到服务器的客户端的完全的连接 / 会话的详细信息。包括“接受 / 发送”的包数量、会话 id 、操作延迟、最后的操作执行等等信息。 dump 列出未经处理的会话和临时节点。 envi 输出关于服务环境的详细信息（区别于 conf 命令）。 reqs 列出未经处理的请求 ruok 测试服务是否处于正确状态。如果确实如此，那么服务返回“ imok ”，否则不做任何相应。 stat 输出关于性能和连接的客户端的列表。 wchs 列出服务器 watch 的详细信息。 wchc 通过 session 列出服务器 watch 的详细信息，它的输出是一个与 watch 相关的会话的列表。 wchp 通过路径列出服务器 watch 的详细信息。它输出一个与 session 相关的路径。 |
今日新闻 |
推荐新闻 |
| CopyRight 2018-2019 办公设备维修网 版权所有 豫ICP备15022753号-3 |