Some times, it looks like the pip installer used by default use the system module folder to store python modules. So, to ensure the correct installation inside the environment, you can call splicitely the pip binary inside the environment location. Something like this:
$ ~/anaconda/envs/[venv_name]/bin/pip install [package_name]
If you'd prefer that conda's base environment not be activated on startup, set the auto_activate_base parameter to false:
$ conda config --set auto_activate_base false