What to do with old glue on the wall?

[Swtp]

I have just removed some fabric backed vinyl paper from some walls and am going to be putting on another fabric backed vinyl. Both the old and my new are textured papers (fine ridges).
The wall now looks like a rustic old plaster job - mostly yellow (the old glue) with splotches of white (the wallboard). If the rest of my house had the rustic look I'd be tempted to just leave this wall. I'm sure some people work hard to get that look. :)
But I don't have a rustic rest of the house. I did already go over the wall with some sand paper to knock off loose chips of glue. I also went over the wall with a sponge and hot water with some cleaner in it. The glue became a bit soft and sticky so I took the opportunity to use a screen sander on to smooth it down even a bit more. The wall is also textured (orange peel) as is the rest of the house. The previous wall paper had just gone right over it and it didn't show through.
My question: Can I just leave that glue and perhaps do a primer/sizer over it and go ahead and paper? Or use a primer/sealer then a sizer and then paper? I can't imagine trying to get the old glue off. Please advise.
Swtp

[B Blanchard]

It's always best to remove the old paste. It's old, dirty, might have some mildew spores in it, etc. You could spray it with some wallpaper remover, let it sit for a little while until it's soft, then scrub it with the nylon pad or scrape it off with a broad knife. Finish up with a sponge and clean water wash down.

Then you could put a coat of wallpaper primer on the clean walls. When that is dry hang the new paper. You won't have to worry about any potential problems.

If you go the route of hanging directly over the old paste don't bother putting a primer of any kind over it. That's asking for problems. Just paste up your new paper and hang directly on the old paste.