Taking a cue from integers, where "factoring" means to write an integer as a product of other integers, we take "factoring polynomials" to similarly mean writing a polynomial as a product of other polynomials.
When one is lucky, factoring integers can sometimes be easy. For example, any integer whose base $10$ form ends in an even digit or $5$ has a factor of $2$ or $5$, respectively. Similarly, any integers whose base $10$ digits sum to a multiple of $3$ will also have a factor of $3$. There are several other "divisibility tricks" one can employ.
However, even with such tricks large integers can prove extremely difficult to factor. As an example, try to factor $137703491$. Discovering this value is the product of just two primes, $7919$ and $17389$, is not an easy task without utilizing technology in some fashion. Interestingly, larger products can prove not only difficult, but downright intractable  even with the help of technology!^{†}
† : Indeed, the security of the internet rests in part on this difficulty. The interested reader might want to read about the RSA Factoring Challenge, where cash prizes from $\$1000$ to $\$200,000$ were offered for correctly factoring individual numbers (the biggest had $617$ digits). Some of these numbers took decades to factor. Many remain unfactored, and will most likely stay that way for decades more.
Perhaps unsurprisingly, given the relationship between polynomials and integers, factoring polynomials behaves similarly  some have factors that are easy to identify. Others can be incredibly difficult to find.
The easiest factor of a polynomial to identify is a monomial factor common to every term. Upon discovering such a common factor, we can use the distributive property in reverse to (at least partially) factor the polynomial.
As an example, if we wished to factor $10x^2y^3z + 4x^2y$, we should note that $10x^2y^3z$ and $4x^2y$ share factors of $2$, $x^2$, and $y$. Thus, $$10x^2y^3z + 4x^2y = 2x^2y(5y^2z + 2)$$ Of course, not every polynomial has a common monomial factor on every term. For such polynomials some other trick or technique will need to be used. In the fortuitous case that the polynomial "fits the form" of a binomial that has been squared, cubed, or raised to some higher integer power, the result of the Binomial Theorem can be easily reversed.
For example, recall the special product rule for the square of a binomial, $(a+b)^2 = a^2 + 2ab+ b^2$. If a polynomial fits the form seen on the right side of this equation, we can immediately conclude it can be factored in a manner that agrees with the left side.
Nicely, checking to see if an expression "fits the form" on the right can be distilled down to a small handful of easilyanswered questions:
As an example, suppose we wish to factor $9x^2 + 12xy^3 + 4y^6$:
We could do something similar to discover a polynomial factors as a cube of a binomial.
We know that $(a+b)^3 = a^3 + 3a^2b + 3ab^2 + b^3$, so when a polynomial fits the form on the right side of this equation, we can factor it immediately into the cube of the binomial associated with the left side. Deciding if it "fits the form" requires more (and slightly more complicated) questions this time, but the overall strategy is the same:
As an example, consider trying to factor $x^6  6x^4y + 12x^2y^2  8y^3$:
We could create similar factoring rules for fourth and higher powers of binomials using the same techniques, but these are even less likely to be applicable to a randomly chosen polynomial and the list of questions needed for each gets longer and more complicated to answer as the exponent on these binomials grows. Thus, there will be a diminishing rate of return to pursuing factoring rules for higher powers of binomials in this way.
Let us consider what might go through our mind as we try to factor something like the following: $$xy^2  2xy + x  y + 1$$ We might initially be excited to see a common factor of $x$ on the first three terms  only to be sadly crestfallen upon realizing the last two terms do not have this common factor!
Still, lets play with those first three terms anyways and see where it leads...
$$xy^2  2xy + x  y + 1 = x(y^2  2y + 1)  y + 1$$ Aha! There's a recognizable piece in there! Notice that $y^2  2y + 1$ fits the form of a perfect square! (the first term is the square of $y$, the last term is the square of $1$, and the middle term is twice the product of these two in magnitude).Running with this, we can $$\begin{array}{rcl} xy^2  2xy + x  y + 1 &=& x(y^2  2y + 1)  y + 1\\ &=& x(y1)^2  y + 1 \end{array}$$ Of course now, we can now more easily see a connection between the $(y1)$ factor and the $y + 1$ sitting on the end. One is almost identical to the other  except the two signs are wrong.
There's an easy fix  we just factor out the negative. This reveals a common factor of $(y1)$ which can be pulled out to yield a successful factorization of our original expression: $$\begin{array}{rcl} xy^2  2xy + x  y + 1 &=& x(y1)^2  (y1)\\ &=& (y1)(x(y1)  1)\\ &=& (y1)(xy  x  1) \end{array}$$
This strategy of "nibbling" at a piece of the expression where there is something we can try and crossing our fingers hoping that the rest of the problem works out can often be successful. Let's consider another example.
Suppose we wish to factor $x^2  2xy + 5x  10y$. Here, factoring out a common $x$ from the first three terms sadly doesn't lead anywhere: $$x^2  2xy + 5x  10y = x(x2y + 5)  10y = ?$$ However, one might also notice that there is a common factor of $5$ in the last two terms. What if we grouped the first two terms together, pulling out their common $x$  and grouped the last two terms together, factoring out the aformentioned $5$: $$\begin{array}{rcl} x^2  2xy + 5x  10y &=& (x^2  2xy) + (5x  10y)\\ &=& x(x2y) + 5(x2y) \quad \scriptsize{\textrm{How nice was that! There's a common $(x2y)$ factor!}}\\ &=& (x2y)(x+5) \end{array}$$
This technique used above of splitting a polynomial into groups of terms, and then identifying a factor common to all of the groups can be helpful in factoring polynomials more generally, as the following two additional examples suggest.
Importantly, though  don't be afraid to "play" with expression in question.You may have to consider writing an expression in several different ways before something breaks. Not everything will work out beautifully on your first attempt. Be persistant though  "nibble hopefully"  and something will eventually crack!
$$\begin{array}{rcl} x^2 + 3x  3y  y^2 &=& (x^2  y^2) + (3x  3y) \quad \scriptsize{\textrm{notice we had to move some terms around}}\\ &=& (x+y)(xy) + 3(xy)\\ &=& (xy)(x+y+3)\\\\ a^2b + 2a^2c + ab + 2ac + b + 2c&=& (a^2b + ab + b) + (2a^2c + 2ac + 2c) \quad \scriptsize{\textrm{again we moved some terms around}}\\ &=& b(a^2 + a + 1) + 2c(a^2 + a + 1)\\ &=& (a^2 + a + 1)(b + 2c) \end{array}$$Finding products of polynomials often results in a number of "like terms" (i.e., terms that differ only in their coefficients) that we then collect together. As an example, $$\begin{array}{rcl} (x+3)(x^2 + x + 5) &=& x(x^2+x+5) + 3(x^2+x+5)\\ &=& x^3 + x^2 + 5x + 3x^2 + 3x + 15\\ &=& x^3 + (x^2 + 3x^2) + (5x + 3x) + 15 \quad \quad {\tiny \textrm{here, we show the liketerms collected grouped together}}\\ &=& x^3 + 4x^2 + 8x + 15 \end{array}$$ With this in mind, note that factoring the last expression by grouping is indeed possible  but doing so requires some significant and very nonobvious inspiration strike first!
Think about the steps just shown, but in reverse: $$\begin{array}{rcl} x^3 + 4x^2 + 8x + 15 &=& x^3 + (x^2 + 3x^2) + (5x + 3x) + 15 \quad \quad {\tiny \textrm{we can separate $4x^2$ and $8x$ this way, but it's not yet obvious this will help!}}\\ &=& x^3 + x^2 + 5x + 3x^2 + 3x + 15 \quad \quad {\tiny \textrm{we can rearrange the terms this way, but again  this is not very obvious}}\\ &=& x(x^2+x+5) + 3(x^2+x+5) \quad \quad {\tiny \textrm{finally, if we group into groups of 3 we see factoring by grouping can work!}}\\ &=& (x+3)(x^2 + x + 5) \end{array}$$
We could have split the $4x^2$ and $8x$ in other ways, but most of these result in factoring by grouping not being fruitful. Consider for example, using $4x^2 = 2x^2 + 2x^2$ and $8x = x + 7x$  go ahead and try it; you won't see a common factor form. The same can be said of different rearrangements of terms  most will lead to a failed attempt at factoring by grouping.
The above illustrates two important things to keep in mind about factoring by grouping: 1) if the polynomial factors, there is a way to split and rearrange terms, upon which factoring by grouping will actually be successful; but 2) the number of possibilities you have to check with regard to all those ways to split and rearrange terms can be so large that exploring all the options in order to find the one that works is impossible in any practical sense.
Still, when the polynomials are small enough we can sometimes navigate those possibilities in a reasonable amount of time. This is especially true when we suspect the polynomial in question factors into two binomials  as is often the case with trinomials, for example.
To see this, let us mimic the work we just did above, but this time let us start with a simpler product, one of two binomials. Then, we'll reflect on those calculations and how we might produce them (and consequently, a factoring of the associated trinomial) by "working backwards"  first in our specific example, and then more generally.
With that in mind, note that the following is certainly true: $$\begin{array}{rcl} (2x+3)(x7) &=& 2x(x7) + 3(x7)\\ &=& 2x^2 14x + 3x  21\\ &=& 2x^2 11x  21 \end{array}$$
Going backwards, notice that we need to split $11x$ into the sum of terms, $(14x + 3x)$. What is special about these two terms? How do we know these are the two we need to sum if we start with $2x^2 11x  21$? Notice that in the factored form of the trinomial above, the $14x$ came from $(2x)(7)$ and the $3x$ came from $(3)(x)$. In the trinomial being factored however, notice that $2x^2 = (2x)(x)$ and $21 = (3)(7)$. The same expressions are involved, but they have "switched partners".
Now  given that the leading coefficient of $2$ and the constant term $21$ have a small number of factors each  there are only a few ways this could happen. The squares below give us a convenient way check the various possibilities. Note that for each, we insist that the row products are $2x^2$ and $21$, respectively), but seek column products (whose factors have "switched partners" from those forming the row products) that sum to $11x$. $$\require{color} \begin{array}{cc}\hline 2x & x \\\hline 1 & 21 \\\hline \end{array} \hspace{0.1in} \begin{array}{cc}\hline 2x & x \\\hline 1 & 21 \\\hline \end{array} \hspace{0.1in} \begin{array}{cc}\hline 2x & x \\\hline 3 & 7 \\\hline \end{array} \hspace{0.1in} \begin{array}{cc}\hline 2x & x \\\hline 3 & 7 \\\hline \end{array} \hspace{0.1in} {\color{blue} \begin{array}{cc}\hline 2x & x \\\hline 7 & 3 \\\hline \end{array}} \hspace{0.1in} \begin{array}{cc}\hline 2x & x \\\hline 7 & 3 \\\hline \end{array} \hspace{0.1in} \cdots$$ Notice that in the blue square, we find what we were looking for  the column products sum to $14x + 3x = 11x$.
The rest follows immediately from factoring by grouping  no rearranging of the terms summing $11x$ is required, as in either order factoring by grouping leads to a correct factorization.
To be even more efficient, we can initially ignore the signs of the factors and focus solely on their magnitudes. Of course, this comes at the cost of having to pay attention to whether the sum or the difference of the column totals gives us the middle term sought. Then, one still needs to make sure appropirate signs can be inserted.
As an example, suppose we wish to factor $12x^2 + 5x  2$: First, we start listing squares with row products of $12x^2$ and $2$, looking for one with a sum or a difference of column products equal to $5x$: $$\begin{array}{cc}\hline x & 12x \\\hline 1 & 2 \\\hline \end{array} \hspace{0.1in} \begin{array}{cc}\hline 2x & 6x \\\hline 1 & 2 \\\hline \end{array} \hspace{0.1in} {\color{purple} \begin{array}{cc}\hline 3x & 4x \\\hline 1 & 2 \\\hline \end{array}} \hspace{0.1in} \cdots$$ We stop on the purple one above as the column products $3x$ and $8x$ can be subtracted in some order to make $5x$. Now we play with the signs to get $5x$ as the sum of the column products. Recall, we need a row product of $2$ in the second row  so exactly one of its elements must be negative and the other positive: $${\color{blue} \begin{array}{cc}\hline 3x & 4x \\\hline 1 & 2 \\\hline \end{array}}$$ Thus, $$\begin{array}{rcl} 12x^2 + 5x  2 &=& 12x^2 + 8x  3x  2\\ &=& 4x(3x + 2)  (3x + 2)\\ &=& (3x+2)(4x1) \end{array}$$ Of course, the attentive reader will notice that one can simply look at the diagonals of the blue square above and bypass the factoring by grouping steps altogether!
One should be aware however  as fun as writing out squares of factors can be  when the numbers involved have many factors, this can be a tedius process! We really need a better way than what amounts to a careful organized "trialanderror" approach. Thankfully, for any polynomial of the form $ax^2 + bx + c$, there is a much, much better way! ..one that requires no guessing or careful checking of different cases at all! This other method will be more natural to introduce a bit later, so we will keep the above as our method for the moment (with a promise that until then, the number of factors for any problems considered won't get too large).
We can also use factoring by grouping to establish additional rules for other, often seen forms for an expression. One of these is known as a difference of squares, and has the form $x^2  y^2$ for some $x$ and $y$.
Given that $x^2y^2$ consists of only two terms with nothing apparently in common, we might be initially befuddled as to how to split things into groups whereby something common can be "pulled out". Following the oftused strategy of "put in what you want or need, and then compensate for the insertion", we will add new term of our choosing and then immediately subtract it, so as to essentially add a wellchosen value of zero, which leaves the original expression effectively unchanged. Realizing it would be desirable to have a factor of $x$ in common with the first term $x^2$ and a factor of $y$ in common with the second term $y^2$, let us try adding and subtracting $xy$: $$\begin{array}{rcl} x^2  y^2 &=& x^2 + \overbrace{xy  xy}^{zero}  y^2\\ &=& (x^2 + xy) + (xy  y^2) \quad \scriptsize{\textrm{grouping the first two and last two terms}}\\ &=& x(x+y) + (y)(x+y) \quad \scriptsize{\textrm{after factoring each group}}\\ &=& (x+y)(x + (y)) \quad \scriptsize{\textrm{the common } (x+y) \textrm{ is factored out}}\\ &=& (x+y)(xy) \end{array}$$
Thus, we have deduced the difference of squares special product rule:
$x^2  y^2 = (x+y)(xy)$ 
Just like the special product rules for squares and cubes of binomials that we examined at the begining this section, the variables $x$ and $y$ above can function as "standins" for other expressions. As two quick examples, note that $$\begin{array}{rcll} w^4  9 &=& (w^2 + 3)(w^2  3) &\quad \quad \textrm{Here, $x=w^2$ and $y=3$}\\ p^2 + 2pq + q^2  4 &=& (p+q+2)(p+q2) &\quad \quad \textrm{This time, $x=(p+q)$ and $y=2$} \end{array}$$
Now that we know how to factor a difference of squares  what about the difference of cubes, $x^3y^3$?
Taking a cue from how adding a wellchosen value of zero helped us factor a difference of squares by grouping, let us apply a similar strategy, as seen below:
$$\begin{array}{rcl} x^3  y^3 &=& x^3 + \overbrace{x^2y + x^2y}^{\textrm{zero}}  y^3\\ &=& (x^3  x^2y) + (x^2y  y^3)\\ &=& x^2(xy) + y(x^2y^2)\\ &=& x^2(xy) + y(x+y)(xy)\\ &=& (xy)(x^2 + y(x+y))\\ &=& (xy)(x^2 + xy + y^2) \end{array}$$You may have been surprised by the term added and then subtracted above. However, one will note that adding and subtacting $xy$ runs into problems that $x^2y$ overcomes.
The sum of cubes, $x^3 + y^3$, can be factored using a very similar process. The upshot of these pursuits is two new special product rules that can be used to factor polynomials that fit the following forms:
$$\boxed{\displaystyle{\begin{array}{rcl} x^3  y^3 &=& (x  y)(x^2 + xy + y^2)\\ x^3 + y^3 &=& (x + y)(x^2  xy + y^2) \end{array}}}$$Just as before, you should think of the above as templates where the $x$ and $y$ above can be any expressions. For example, if one was attempting to factor $a^3b^6 + 8$, one might notice that the first term is the cube of $ab^2$ and the second is the cube of $2$. Substituting these expressions in the above rules as our $x$ and $y$ respectively, we find
$$\begin{array}{rcl} a^3b^6+8 &=& (ab^2)^3 + 2^3\\ &=& (ab^2 + 2)((ab^2)^2  (ab^2)(2) + (2)^2)\\ &=& (ab^2 + 2)(a^2b^4  2ab^2 + 4) \end{array}$$We now have a number of tools to utilize when attempting to factor a polynommial. Some of these are easier to apply than others. So in the interest of not wasting time when factoring, one should always try the easiest methods to apply first!
As such, when attempting to factor something, ask yourself the following questions in this order:
Is there a common monomial factor on all the terms?
Can one of the special product rules be used? Does it fit one of the following forms?
Could it possibly be the product of two binomials? (if so, try finding a "square of 4 things" that works out)
Does factoring by grouping work "as is"? (try grouping different numbers of terms, but in the order they were given)
Does factoring by grouping work after rearranging terms?
Can you cleverly split terms and rearrange to get factoring by grouping to work? (don't spend too much time on this approach  it can lead you down many "false paths")
The above spends a good bit of effort developing several techniques/tricks that can help when trying to factor a polynomial  but is it possible for a polynomial to not be able to be factored any further? ...and how would we know that is the case?
We should be careful about expressing exactly what we mean here. As a matter of verbiage, we say the degree of a polynomial of one variable is the highest exponent seen on the variable in question in the polynomial. So for example, $x^2 + 3x  5$ has degree 2, while $2x^3  x + 7$ has degree $3$.
As an unrelated side note, the degree of a term in a polynomial of one variable is the exponent on the variable seen in that term. Also, the traditional way to write a polynomial (unless there is an advantage in doing something else) is to order the terms of the polynomial in descending order by the degree of the terms involved  much like the digits of the related base $x$ integer.
Let us call a polynomial with integer coefficients irreducible when it can't be factored into a product of two other polynomials with integer coefficients each of a lesser degree.
So expressing things more clearly, we are curious if it is possible a polynomial can be irreducible and how can we tell when that is the case?
The answer to the first question is a resounding yes. For example, trying to factor $x^2+x+1$ into $(x+r)(x+s)$ where $r$ and $s$ are integers, as described in the last section, quickly leads nowhere.
The answer to the second question however, is more complicated. The following result, named after the German mathematician Gotthold Eisenstein gives us a partial answer:Eisenstein's Criterion: A polynomial $a_n x^n + a_{n1} x^{n1} + \cdots + a_2 x^2 + a_1 x + a_0$ with integer coefficients is irreducible when there is a prime $p$ where:

Sadly, this is only a sufficient and not a necessary condition for a polynomial to be irreducible. That means that Eisenstein's criterion not holding does not guarantee the polynomial can be factored into the product of two polynomials each of lesser degree. Still, its better than nothing.
Let's see how it works...
Suppose a polynomial $a_n x^n + a_{n1} x^{n1} + \cdots + a_2 x^2 + a_1 x + a_0$ that satisfied Eisenstein's criterion could be factored into two polynomials of lesser degree. (Our aim is of course to show such an assumption leads to some contradiction, and thus this can't every actually happen.). In particular, let us assume that $$a_n x^n + a_{n1} x^{n1} + \cdots + a_2 x^2 + a_1 x + a_0 = (c_r x^r + c_{r1} x^{r1} + \cdots + c_0)(d_s x^s + d_{s1} x^{s1} + \cdots + d_0)$$ Note it must be the case that $a_0 = c_0 d_0$. So if $a_0$ is evenly divisible by $p$, but not $p^2$, one of $c_0$ and $d_0$ must be evenly divisible by $p$, but not both. Without loss of generality, let us assume $p$ divides $c_0$ evenly but not $d_0$.
It must also be the case that $a_n = c_r d_s$. Given that $p$ does not evenly divide $a_n$, it can neither divide $c_r$ nor $d_s$.
Now, note that $a_1 = c_0 d_1 + c_1 d_0$. As $p$ divides $a_1$ and $c_0$, but not $d_0$, it must be that $p$ divides $c_1$.
Likewise, note that $a_2 = c_0 d_2 + c_1 d_1 + c_2 d_0$. As $p$ divides $a_2$, $c_0$, and $c_1$, but not $d_0$, it must be that $p$ divides $c_2$.
Similarly, note that $a_3 = c_0 d_3 + c_1 d_2 + c_2 d_1 + c_3 d_0$. As $p$ divides $a_3$, $c_0$, $c_1$, and $c_2$, but not $d_0$, it must be that $p$ divides $c_3$.
We can continue in this way to establish that $p$ divides $c_4, c_5, c_6, \ldots$ up to $c_r$. However, this last conclusion that $p$ divides $c_r$ is a problem! Recall, we earlier concluded that $p$ didn't divide $c_r$ evenly! We have found our contradiction.
Thus, if Eisenstein's criterion holds, no such polynomials of lesser degree exist, and $a_n x^n + a_{n1} + \cdots + a_2 x^2 + a_1 x + a_0$ is irreducible.
As an example of how to use Eisenstein's criterion, consider the polynomial $3x^4 + 15x^2 + 10$. Note that the prime $p=5$ evenly divides the last two coefficients, but not the first, and $p^2 = 25$ does not divide the last coefficient. Thus, this polynomial is irreducible and can't be factored into the product of smaller degree polynomials.
As an interesting piece of trivia, at the young age of twenty Eisenstein met the great mathematician William Rowan Hamilton in Dublin, who gave him a copy of his book on Niels Henrik Abel's famed impossibility theorem. Stay tuned, we will have much more to say about this really important theorem later!