Sunday, June 26, 2016

Eclipse 4.5.0 fails to load properly on Ubuntu 16.04

It seems like Eclipse, Spring Source Tool Suits (STS) or any other IDE built on top of Eclipse fail to load properly (frozen, slow, 100% CPU, etc) on Ubuntu 16.04.
You are not alone... GTK compatibility issues.
You will need to start Eclipse with GTK 2, just edit the eclipse.ini file and add the following:
--launcher.GTK_version
2
before the --launcher.appendVmargs line


Saturday, July 4, 2015

ClassCastException on remote EJB call between EARs in Websphere 8.5

We have 2 EARs running on the same Websphere 8.5 Application Server. Each EAR is running in its own classloader. We have a service that initiates a remote EJB call to another service exists in another EAR. Stubs are being generated automatically on startup since WAS 7.
One of our teams tried to perform some clean up that we place in the other EAR that holds our interface in order to have a better isolation in the classpath.
Right after this classpath clean up, we were trying to initiate a remote EJB call, but for some reason it thrown the following exception:
"...ClassCastException...unable to load ..._Stub..."
We didnt understand why this exception came up? why ClassCastException?
After some time, we realized that we had Remote interfaces which were not identical between the EARs and that there were missing classes in classpath. So whenever you have such an architecture, you should make sure that your remote interfaces are identical and you dont have any missing classes in the classpath of the calling EAR.

Tuesday, March 31, 2015

getpocket.com Delicious (del.icio.us) bookmark importer

If you have ever tried to import all/part of your delicious links into getpocket.com, you probably had difficulties using the tool provided by Pocket. Pocket provides a tool that enables you to import your delicious bookmark but it doesnt work well and for some reason, large amount of your private and/or public links are not being imported. I've found out that it could be a parsing issue because of the exported delicious HTML file structure.
An important note by Pocket: Pocket is not a replacement for archival type bookmarking. Your list is a collection of links that you intend to view later. We strongly advise against importing thousands of items here.

Friday, March 20, 2015

Tuesday, March 10, 2015

RUNNABLE thread - It just looks normal

Its been a very interesting day. I just solved a bug that we were chasing after for quite some time. Generally, we collect failed HTTP requests to certain web services into a dedicated Message Queue. We process the messages in parallel by adding them into a fixed Thread Pool.
It worked successfully 99% of the time and happens that several threads have been started and unfortunately didn't accomplish their job.

We couldn't reproduce the issue and it could happen once a week at 5am :)

I couldn't find any exceptions in the server logs and the severs state were just fine. Performance was not affected, RAM & CPU state were normal.
By reading the logs, I knew that the thread hangs during an HTTP request but I knew that whenever there is no response it returns with the appropriate HTTP status code (i.e. 500).

In order to drill down a bit, I wanted to take a look at the JVM of the specific process and for that purpose I used jVisualVM. I couldn't identify anything special - memory state was fine, couldn't identify any deadlocks, parking threads, etc. so decided to generate a Thread Dump:



then passed through the above thread which is in a RUNNABLE state while infinitely waiting for the socket reader to finish its job.
It turn out that even though you set timeout properly (connection and socket timeouts), it can still hang in an infinite state. Since we are using the Google http client API to access several Google services, I couldnt get into the underlying connection manager to cancel the request so I had to wrap the request in a single Future<V> thread with a reasonable timeout that can force an abort to existing request and add the failed message back into the Message Queue.

Finally its all stable again...hmmm lets deal with the next challenge :)



Thursday, December 25, 2014

MySQL server - Lost connection to server during query

There are many reasons for such an error, thats why MySQL wrote a dedicated subsection at their reference manual (http://dev.mysql.com/doc/refman/5.6/en/gone-away.html).
It happened to me just yesterday while trying to query a large table. 
The way I debugged it was by setting the following: 
1. Log level to 2:  --log-warnings=2 
2. Turn on the slow queries log: slow-query-log=1
Running the query again, revealed the following error in MySQL log file: 
"...Got timeout writing communication packets"

Make sure that your timeout parameters were not set in your MySQL configuration file - their default is: 480 hours which is enough:
Interactive_timeout 
wait_timeout
Set the following parameters to 28800 seconds - the default is: 60 seconds which is very low for such a query: 




Sunday, October 12, 2014

MySQL Cluster - Migrating a large InnoDB table to NDB Cluster



If this is the first time you try to migrate a large InnoDB table into a NDB cluster, most probably you will run into some issues.
By nature, both storage engines (InnoDB and NDBCluster) have many differences - its highly recommended to read the following:
It really depends on your MySQL cluster architecture, setup and the amount of data you are trying to migrate. But in general, you will probably run into common issues:
1. The table 'table_1' is full
2. Lock wait timeout exceeded; try restarting transaction

These errors can be related to many different configuration issues, but usually you need to consider setting the following important config.ini parameters:
1. DataMemory and IndexMemory - 
The DataMemory value is in bytes and defines the available space to store database records. Note that this will allocate the entire amount specified in your config file, so make sure that you have the required actual space. 
IndexMemory value is in bytes as well and in case you got index, you will need to define this parameter. It controls the amount of storage used to hash the MySQL cluster indexes - depends on your cluster configuration. Lets say you got 4 Nodes (fragments), 2 replicas and there are 1M rows, you can calculate the size using the following formula (see MySQL documentation):

size  = ( (fragments * 32K) + (rows * 18) ) * replicas

2. Lock wait timeout error can be caused by many different configuration issues, but usually its related to the TransactionDeadlockDetectionTimeout parameter. 
Setting the timeout will require you to set the MaxBufferedEpochs parameter as well.
NOTE: while setting your TransactionDeadlockDetectionTimeout, its highly recommended to make sure that your mysqld innodb_lock_wait_timeout set to the correct timeout (http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html).


I suggest to read the following blog post as well: 


Finally, don't forget to delete the ndbd_mgm config bin (cache file located in the config location) each time you make changes to your config.ini.


Good luck
Eran