What’s difficult? The most difficult in starting from mobile view, in my opinion, is lack of clear visual design concept. This issue comes strictly from creative, artsy perspective, in which the mobile version sometimes IS a deficient design. Yes, that’s the truth – there is not much space for expression and letting unhampered imagination speak, especially if the design isn’t pure and minimalistic. There is a lot to care about composition of color and shape which has to fit in small space and still look inviting and clear, not to mention the intuitive handling.
What’s the solution? I would recommend sketching all of the versions with as many visual ideas as possible and then start the mobile option from a technical and pure user experience and front-end developing view. It seems the best answer for the quandary, a compromise to get only pros from both perspectives!