Saturday, August 31, 2013

JDBC broken pike error

Error

 [2013-08-31 05:41:15,462] [ERROR] [SimpleAsyncTaskExecutor-2] [TaskletStep] [438]: JobRepository failure forcing exit with unknown status
org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL [UPDATE BATCH_STEP_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE STEP_EXECUTION_ID = ?]; Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2689)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2618)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1551)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:2978)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:2902)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:930)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1159)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1076)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1061)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:106)
        at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:824)
        at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:818)

Cause:
MySQL connections are getting terminated by MySQL before they have been released by the connection pool. Then the next time you try to get a connection from the pool, you'll get a dead connection which will give a "broken pipe" error, as in the above stack trace.

Resolution

Evict the idle session before MySQL does. Retrying the connection, will simply create a new connection. The following parameters can be added to the Jira datasource connection URL in Confluence's server.xml for the same:
minEvictableIdleTimeMillis=21600000 (evict after 6 hours of inactivity, given that MySQL has a default timeout of 8 hours for idle database connections after which it closes the connection)
timeBetweenEvictionRunsMillis=1800000 (run the evicter every 30 minutes)
numTestsPerEviction=-1 (check every connection)