Bug: hadoop map reduce jobs are pending too long.

It seems that one and the same problem presents in both Hortonworks SandPit virtual machine and HDInsight Emulator (most likely because of old versions of Hadoop): jobs are always in pending status and no explicit errors/exceptions appear in jobtracker/tasktracker logs.

The solution for the Emulator is to include the next section into the Hadoop configuration file:

  <description> Maximum physical memory tasktracker should reserve for mapreduce tasks.
  If tasks use more than the limit, task using maximum memory will be killed.
   Expert only: Set this value if tasktracker should use a certain amount of memory
   for mapreduce tasks. In MapR Distro warden figures this number based
   on services configured on a node.
   Setting mapreduce.tasktracker.reserved.physicalmemory.mb to -1 will disable
   physical memory accounting and task management.

For emulator-based installation the config is stored here: C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\mapred-site.xml.


About fdtki

Sr. BI Developer | An accomplished, quality-driven IT professional with over 16 years of experience in design, development and implementation of business requirements as a Microsoft SQL Server 6.5-2014 | Tabular/DAX | SSAS/MDX | Certified Tableau designer
This entry was posted in Administration and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s