依赖来源：httpasyncclient-4.1.4.jar

现象

程序报错too many open files
线程数飙升、句柄数飙升
thread dump显示大量

"I/O dispatcher 7215" #9102 prio=5 os_prio=0 tid=0x00002b7ba036a800 nid=0x6f24 runnable [0x00002b7d98d41000]java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked <0x0000000778c3e978> (a sun.nio.ch.Util$3)- locked <0x0000000778c3e968> (a java.util.Collections$UnmodifiableSet)- locked <0x0000000778c3e850> (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)at java.lang.Thread.run(Thread.java:748)

原因

RestHighLevelClient底层依赖的是CloseableHttpAsyncClient的实现类CloseableHttpAsyncClientBase

abstract class CloseableHttpAsyncClientBase extends CloseableHttpPipeliningClient {private final NHttpClientConnectionManager connmgr;private final Thread reactorThread;public CloseableHttpAsyncClientBase(final NHttpClientConnectionManager connmgr,final ThreadFactory threadFactory,final NHttpClientEventHandler handler) {super();this.connmgr = connmgr;if (threadFactory != null && handler != null) {this.reactorThread = threadFactory.newThread(new Runnable() {@Overridepublic void run() {try {final IOEventDispatch ioEventDispatch = new InternalIODispatch(handler);connmgr.execute(ioEventDispatch);} catch (final Exception ex) {log.error("I/O reactor terminated abnormally", ex);} finally {status.set(Status.STOPPED);}}});} else {this.reactorThread = null;}this.status = new AtomicReference<Status>(Status.INACTIVE);}@Overridepublic void start() {if (this.status.compareAndSet(Status.INACTIVE, Status.ACTIVE)) {if (this.reactorThread != null) {this.reactorThread.start();}}}@Overridepublic void close() {if (this.status.compareAndSet(Status.ACTIVE, Status.STOPPED)) {if (this.reactorThread != null) {try {this.connmgr.shutdown();} catch (final IOException ex) {this.log.error("I/O error shutting down connection manager", ex);}try {this.reactorThread.join();} catch (final InterruptedException ex) {Thread.currentThread().interrupt();}}}}
}

可以看到调用close时主要是两个操作，调用NHttpClientConnectionManager的shutdown方法并阻塞等待reactorThread结束。reatorThread的run方法中调用了AbstractMultiworkerIOReactor 的execute方法，这个方法由于status值没有改变过，实际上是个死循环。因此该线程一直无法结束。

public abstract class AbstractMultiworkerIOReactor implements IOReactor {@Overridepublic void execute(try {for (;;) {if (this.status.compareTo(IOReactorStatus.ACTIVE) == 0) {processEvents(readyCount);}// Verify I/O dispatchersfor (int i = 0; i < this.workerCount; i++) {final Worker worker = this.workers[i];final Throwable ex = worker.getThrowable();if (ex != null) {throw new IOReactorException("I/O dispatch worker terminated abnormally", ex);}}if (this.status.compareTo(IOReactorStatus.ACTIVE) > 0) {break;}}} catch (final ClosedSelectorException ex) {addExceptionEvent(ex);} catch (final IOReactorException ex) {if (ex.getCause() != null) {addExceptionEvent(ex.getCause());}throw ex;} finally {doShutdown();synchronized (this.statusLock) {this.status = IOReactorStatus.SHUT_DOWN;this.statusLock.notifyAll();}}}```