1 java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: offset (0) + length (8) exceed the capacity of the array: 4

做簡(jiǎn)單的incr操作時(shí)出現(xiàn)，原因是之前put時(shí)放入的是int  長(zhǎng)度為 vlen=4 ，不適用增加操作，只能改為long型 vlen=8

2 寫(xiě)數(shù)據(jù)到column時(shí) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: 10.xx.xx.37:60020,    或是 org.apache.hadoop.hbase.NotServingRegionException: Region is not online:   這兩種出錯(cuò)，master-status中出現(xiàn)Regions in Transition 長(zhǎng)達(dá)十幾分鐘，一直處于PENDING_OPEN狀態(tài)，導(dǎo)致請(qǐng)求阻塞。目前把10.xx.xx.37這臺(tái)機(jī)器下線(xiàn)，運(yùn)行一夜穩(wěn)定，沒(méi)有出現(xiàn)因split造成的阻塞。懷疑是機(jī)器問(wèn)題。Hmaster的日志顯示這臺(tái)region server 不停的open close，不做任何split 或flush

RIT 的全稱(chēng)是region in transcation. 每次hbase master 對(duì)region 的一個(gè)open 或一個(gè)close 操作都會(huì)向Master 的RIT中插入一條記錄,因?yàn)閙aster 對(duì)region 的操作要保持原子性,region 的 open 和 close 是通過(guò)Hmaster 和 region server 協(xié)助來(lái)完成的. 所以為了滿(mǎn)足這些操作的協(xié)調(diào),回滾,和一致性.Hmaster 采用了 RIT 機(jī)制并結(jié)合Zookeeper 中Node的狀態(tài)來(lái)保證操作的安全和一致性.

OFFLINE, // region is in an offline state
PENDING_OPEN, // sent rpc to server to open but has not begun
OPENING, // server has begun to open but not yet done
OPEN, // server opened region and updated meta
PENDING_CLOSE, // sent rpc to server to close but has not begun
CLOSING, // server has begun to close but not yet done
CLOSED, // server closed region and updated meta
SPLITTING, // server started split of a region
SPLIT // server completed split of a region

 進(jìn)一步發(fā)現(xiàn)是load balance的問(wèn)題 region server不停重復(fù)的被open close，參考http:///hbase/book.html#regions.arch.assignment  重啟了region server正常

后來(lái)的代碼運(yùn)行中又出現(xiàn)region not on line ，是NotServingRegionException拋出的，原因是“Thrown by a region server if it is sent a request for a region it is not serving.”

 為什么會(huì)不斷請(qǐng)求一個(gè)離線(xiàn)的region？且這種錯(cuò)誤集中在150個(gè)中的3個(gè)region，追蹤服務(wù)器端log，region 會(huì)被CloseRegionHandler關(guān)掉，過(guò)了20分鐘左右才重新打開(kāi)，關(guān)掉后客戶(hù)端請(qǐng)求的region仍然是這個(gè)關(guān)閉的region？

3 設(shè)置開(kāi)關(guān)不寫(xiě)入hbase并不生效

代碼初上線(xiàn)，增加了開(kāi)關(guān)，萬(wàn)一hbase有問(wèn)題則關(guān)閉掉開(kāi)關(guān)。但是出現(xiàn)問(wèn)題了發(fā)現(xiàn)程序卡死，目前認(rèn)為原因是不斷加長(zhǎng)的retry機(jī)制，60秒超時(shí)，1-32秒的10次retry，萬(wàn)一出問(wèn)題，切換開(kāi)關(guān)也沒(méi)有用。

需要配置rpc超時(shí)參數(shù)和retry time解決它

4 flush、split、compact導(dǎo)致stop-the-world

出現(xiàn)長(zhǎng)時(shí)間的flush split操作導(dǎo)致hbase服務(wù)器端無(wú)法響應(yīng)請(qǐng)求。需要調(diào)整region大小，并測(cè)試獲取flush次數(shù)

5 hbase參數(shù)設(shè)置

hbase.regionserver.handler.count

考慮到sas盤(pán)的io能力，設(shè)置為50

hbase.hregion.memstore.block.multiplier

當(dāng)memstore的大小為hbase.hregion.memstore.flush.size的multiplier倍數(shù)時(shí)，阻塞讀寫(xiě)進(jìn)行flush，默認(rèn)為2

6 region server crush

Regionserver crash的原因是因?yàn)镚C時(shí)間過(guò)久導(dǎo)致Regionserver和zookeeper之間的連接timeout。

Zookeeper內(nèi)部的timeout如下：

minSessionTimeout 單位毫秒，默認(rèn)2倍tickTime。

maxSessionTimeout 單位毫秒，默認(rèn)20倍tickTime。

（tickTime也是一個(gè)配置項(xiàng)。是Server內(nèi)部控制時(shí)間邏輯的最小時(shí)間單位）

如果客戶(hù)端發(fā)來(lái)的sessionTimeout超過(guò)min-max這個(gè)范圍，server會(huì)自動(dòng)截取為min或max，然后為這個(gè)Client新建一個(gè)Session對(duì)象。

默認(rèn)的tickTime是2s，也就是客戶(hù)端最大的timeout為40s，及時(shí)regionserver的zookeeper.session.timeout設(shè)置為60s也沒(méi)用。

改動(dòng)：

1. 將zookeeper集群的tickTime修改為9s，最大的timeout為180s，同時(shí)修改zookeeper.session.timeout為120s，這樣可以避免GC引發(fā)timeout。
2. 添加參數(shù)hbase.regionserver.restart.on.zk.expire為true，改參數(shù)的作用是當(dāng)regionserver和zookeeper之間timeout之后重啟regionserver，而不是關(guān)掉regionserver。

7 代碼問(wèn)題導(dǎo)致死鎖

master慢查詢(xún)?nèi)罩局幸粋€(gè)查詢(xún)達(dá)到了2小時(shí)，最終導(dǎo)致服務(wù)器響應(yīng)變慢，無(wú)法應(yīng)對(duì)大寫(xiě)入。追究原因是getColumns操作一下取出十幾萬(wàn)的數(shù)據(jù)，沒(méi)有做分頁(yè)；更改程序分頁(yè)500條左右，目前沒(méi)有出現(xiàn)問(wèn)題

8 operation too slow

2012-07-26 05:30:39,141 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":69315,"ts":9223372036854775807,"client":"10.75.0.109:34780","starttimems":1343251769825,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"trackurl_status_list","families":{"sl":[{"timestamp":1343251769825,"qualifier":"zzzn1VlyG","vlen":0}]},"row":""}
刪除一行數(shù)據(jù)用了69315s

而且神奇的是row為""，row無(wú)法設(shè)置null進(jìn)去，但可以增加空串。做了一輪測(cè)試
空row-key  刪除不存在的column  耗時(shí) 700ms

空row-key  刪除存在的column  耗時(shí) 5ms

非空row-key  刪除任意的column  耗時(shí) 3ms

不清楚是否是個(gè)bug，也還不知道怎么就傳了個(gè)空row-key進(jìn)去，目前策略為在代碼端避免對(duì)空row-key做操作。

9 responseTooSlow

2012-07-31 17:52:06,619 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":1156438,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3dbb29e5), rpc version=1, client version=29, methodsFingerPrint=-1508511443","client":"10.75.0.109:35245","starttimems":1343727170177,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}

引用hbase說(shuō)明：The output is tagged with operation e.g. (operationTooSlow) if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged (responseTooSlow) and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself.

目前做法是取消了對(duì)某個(gè)key多個(gè)column的批量delete操作避免阻塞，沒(méi)有發(fā)現(xiàn)新問(wèn)題

10 output error

2012-07-31 17:52:06,812 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@61574be4, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"c":["ALL"]},"maxVersions":1,"row":"zOuu6TK"}), rpc version=1, client version=29, methodsFingerPrint=-1508511443 from 10.75.0.151:52745: output error

11 rollbackMemstore問(wèn)題

較頻繁出現(xiàn)這樣的log：
2012-08-07 10:21:49,887 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: rollbackMemstore rolled back 0 keyvalues from start:0 to end:0

方法解釋為：Remove all the keys listed in the map from the memstore. This method is called when a Put has updated memstore but subequently fails to update the wal. This method is then invoked to rollback the memstore.

很奇怪的是開(kāi)始和結(jié)束的index都為0   

方法中循環(huán)： for (int i = start; i < end; i++) {  

因此是空數(shù)據(jù)，空回滾。需要進(jìn)一步調(diào)查

12 新上線(xiàn)一個(gè)region server  導(dǎo)致region not on line

往錯(cuò)誤的region server服務(wù)器請(qǐng)求region

13 請(qǐng)求不存在的region，重新建立tablepool也不起作用

請(qǐng)求的時(shí)間戳 1342510667

最新region rowkey相關(guān)時(shí)間戳 1344558957

最終發(fā)現(xiàn)維持region location表的屬性是在HConnectionManager中

get Get，delete Delete，incr Increment 是在 ServerCallable類(lèi) withRetries處理

   情景1 若有出錯(cuò)（SocketTimeoutException ConnectException RetriesExhaustedExcetion），則清理regionServer location

   情景2 numRetries 若設(shè)置為1 ，則 循環(huán)只執(zhí)行一次，connect(tries!=0) 為connect(false),即reload=false，不會(huì)進(jìn)行l(wèi)ocation更新，當(dāng)為numRetries>1的時(shí)候才會(huì)重新獲取

get Gets List, put Put或Puts List，delete Deletes List 則調(diào)用HConnectionManager的 processBatch去處理，當(dāng)發(fā)現(xiàn)批量get或者put、delete操作結(jié)果有問(wèn)題，則刷新regionServer location

設(shè)置 numRetries為>1次， 我這里是3次，解決問(wèn)題

14 zookeeper.RecoverableZooKeeper(195): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

這是在我單機(jī)做測(cè)試時(shí)出現(xiàn)的，無(wú)論是從ide或是bin啟動(dòng)hbase，從shell里可以正常連接，從測(cè)試程序中無(wú)法連接，zookeeper端口是2181,客戶(hù)端端口應(yīng)該與zookeeper無(wú)關(guān)才對(duì)，

最終更改配置21818端口換為2181 運(yùn)行正常，應(yīng)該是單機(jī)環(huán)境才要做這種更改。

<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>Property from ZooKeeper's config zoo.cfg.
    The port at which the clients will connect.
    </description>
  </property>