Problems with RabbitMQ
Introduction
RabbitMQ is highly stable subsystem that generally works reliably without any administrative oversight whatsoever. If encountered problems with either subsystem it is almost always following a software upgrade, a database restore, or a server migration. Furthermore, the culprit is almost always Celery.
Diagnosing
The below are some symptom you could face in your edx environment which are regarding RabbitMQ.
The new user registration screen appears to die, and becomes unresponsive after clicking the signup command button. The new user data is never saved into the system and the new user never receives an activation email.
The screen appears to freeze or die after you click the password reset button.New users do not receive their new user activation email.
The drag & drop function appears to work, however the changed value is not recognized. Additionally you cannot save results.
The screen appears to die after submitting a response to an exercise or quiz problem.
The screen appears to die, and the document is never uploaded. The system provides neither a success nor a failure message.
If you’re experiencing any of these symptoms then you’ll next want to review the Open edX application logs for both the LMS and CMS to look for errors.
cat /edx/var/log/lms/edx.log -n 50
cat /edx/var/log/cms/edx.log -n 50
Change user permissions
If the source of your Celery problem is permissions then this will eliminate the problem.
sudo rabbitmqctl set_permissions -p / celery ".*" ".*" ".*"
sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
sudo rabbitmqctl set_permissions -p / edx ".*" ".*" ".*"
After completing the above step restart Rabbitmq
sudo service rabbitmq-server restart
Reset the celery password
In order to reset the password for rabbitmq user we have to find the default password first which is stated in my-passwords.yml file (located at /home/ubuntu). Once you successfully find the default password for all the user use the below command to reset the password.
sudo rabbitmqctl change_password celery YourPasswordForTheCeleryUser
sudo rabbitmqctl change_password edx YourPasswordForTheEdxUser
sudo rabbitmqctl change_password admin YourPasswordForTheAdminUser
After completing the above step restart Rabbitmq
sudo service rabbitmq-server restart
Delete and recreate user
In order to delete RabbitMq user including User, EDX or Celery use the below command.
sudo rabbitmqctl delete_user celery
sudo rabbitmqctl delete_user admin
sudo rabbitmqctl delete_user edx
After deleting the user recreate the user by using following commands (also require default password)
sudo rabbitmqctl add_user celery YourPasswordForTheCeleryUser
sudo rabbitmqctl add_user admin YourPasswordForTheAdminUser
sudo rabbitmqctl add_user edx YourPasswordForTheEdxUser
Once you have successfully created the user now change the permission.
After changing the permission change the user tag
sudo rabbitmqctl set_user_tags celery administrator
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl set_user_tags edx administrator
Test User
To test if all the users are working in rabbitmq use the below commands.
sudo rabbitmqctl authenticate_user celery YourPasswordForTheCeleryUser
sudo rabbitmqctl authenticate_user admin YourPasswordForTheAdminUser
sudo rabbitmqctl authenticate_user edx YourPasswordForTheEdxUser
Restart the platform
After completing we need to restart the open edx platform including all services.
sudo /edx/bin/supervisorctl restart all