Integration with a projector
This page describes the patent-pending Upoint embodiment,
supplementing the Utility Patent, and makes
specific use of a projector.
[00177]
It may be advantageous to put some parts disclosed in this
application in a known relation to one another. For example, projection device
40 and base station 30 may advantageously be incorporated in one and the same
physical device, or be in a fixed relationship to one another. Alternatively,
projection device 40 may be equipped with a coordinate sensing device 201.
Whatever means are used, the system may be set up such that the position and/or
orientation of some components, such as projection device 40 (and its
components) with respect to coordinate system x y z is fixed, known, or can be
measured. If additional appropriate sensors are included (such as, for example,
level sensing device 303, visible markers 302, etc), even the position and/or
orientation of moveable parts of projection device 40 with respect to coordinate
system x y z may be considered measurable and hence known. In particular, such
an integrated device may furthermore incorporate a device like pointing device
20, associated with a pointing line 21; that is, it may be equipped with a
device that allows the orientation and/or position with respect to coordinate
system x y z of a pointing line 21 to be known, measured and/or adjusted
(potentially automatically). This pointing line may, for example, coincide with
the optical axis 401 of projection device 40. In this application, optical axis
401 is defined as the line substantially connecting the optical origin 402 of
the projection beam (see Fig. 23) with the projected center of computer screen
image 50, as projected by projection device 40. (Note that the projected center
of computer screen image 50 may coincide with the center of projection image 70,
notably so if both are rectangular.) Alternatively, the pointing line may
substantially coincide with another line connecting a characteristic point of
projected image 70 (for example, one of its corners) to the optical origin 402
of projection device 40, or with other convenient lines. An even more
advantageous embodiment may result when this integrated device also incorporates
distance measuring device/image capture device 206. To explain, it will now be
assumed that the origin of the x y z system coincides with the origin of the
x’ y’ z’ system of the pointing device 20 that is integrated with base
station 30 and projection device 40, as well as with the optical origin 402 of
the projection beam of projection device 40. It will also be assumed that the
aspect ratio is known of a substantially rectangular computer screen image 50
that is to be projected by projection device 40. If one furthermore assumes that
projection device 40 is aligned in such a way that its optical axis 401 is
substantially perpendicular to (a flat) projection region 60 and that its
optical characteristics (such as the dimensions of its projection beam and the
size of the to-be-projected image) are known, it will be clear to those skilled
in the art that knowledge of the distance between the optical origin of
projection device 40 and (the center of) projection image 70 is sufficient to
uniquely establish the position, shape, orientation and size of projection image
70 (and interaction region 71 and interaction structure 72, assumed in this
example to substantially coincide with projection image 70), as measured with
respect to the x y z coordinate system. Notably, in such a scenario there would
not be a need for the use of a user-wielded pointing device 20 to aid in
establishing the above parameters. Indeed, under these circumstances a
user-wielded pointing device 20 may be used as a direct-pointing instrument
straight-away, without user-assisted calibration, after the above described
automated calibration procedure has been executed so as to establish all
necessary parameters of projection image 70 (and interaction region 71 and
interaction structure 72).
[00178]
If the
integrated device (formed by at least combining, in the fashion described above,
projection device 40 and base station 30, and possibly a pointing device 20) is
not aligned perfectly perpendicular to projection region 60, an effect commonly
referred to as keystoning will result in a projection image 70 that is not
rectangular. For example, if the optical axis 401 is perpendicular to (a
vertically oriented) projection region 60 in a horizontal but not in a vertical
sense, projection of a rectangular computer screen image 50 will generally
result in a projection image 70 that is an isosceles trapezoid with two
horizontal edges. This is the most common form of image distortion, and is known
as vertical keystoning. Conversely, if the optical axis 401 is perpendicular in
a vertical, but not in a horizontal sense, the resulting isosceles trapezoid
will have two vertical edges; this effect is known as horizontal keystoning. If
both horizontal and vertical misalignments are not too severe, the assumption
that projection image 70 (and interaction region 71 and interaction structure
72) is rectangular may still result in a system that affords the feeling of
direct pointing, even if the system also assumes the vertical and horizontal
alignments are perfectly perpendicular; the reasons for this are explained in
paragraph 91.
[00179]
In the
more general case, as depicted in prior art Fig. 22, one or both of these
misalignments may be substantial. In such a case, the keystone effect will
result in a projection image 70 that is neither rectangular nor shaped like an
isosceles trapezoid. As explained in numerous prior art (such as US patent
6,520,647 to Mitsubishi Electric Research Laboratories Inc., and US patent
6,877,863 to Silicon Optix Inc.), vertical keystoning can easily be corrected
for by incorporating sensors like level sensing device 303 – see also prior
art Fig. 22. Using such sensors, the vertical angle between the (vertically
oriented) projection image 70 and optical axis 401 can easily be established.
If, furthermore, distance measuring device 206 is subsequently used to
substantially establish the distance between the projected version of the center
of computer screen image 50 and the optical origin 402 of projection device 40
(i.e., measured along the optical axis 401), those skilled in the art will
appreciate that inclusion of information on the optical characteristics of
projection device 40 will, once again, provide sufficient information to be able
to establish position, shape, orientation and size of an interaction structure
72 that is near-coincident with an interaction region 71 and projection image
70, provided the horizontal keystone effect is not too severe. Note that this
may be done independent of whether or not the vertical keystone effect is
corrected for.
[00180]
If the
horizontal keystone effect is also severe, the above cited prior art teaches
that additional sensors may be used to establish the horizontal angle between
projection region 60 and optical axis 401, and that they may be used to correct
for the horizontal keystone effect. Using both vertical and horizontal keystone
correction, a situation as depicted in Fig. 21 may thus arise, characterized by
a non-perpendicular alignment both in horizontal and vertical sense, nonetheless
resulting in an substantially undistorted rectangular projection image 70. Such
sensors may be devices such as used in distance measuring device / image
capturing device 206 – see also prior art Fig. 22. Alternatively, distance
measuring device 206 may be used to establish the distance from the optical
origin 402 of the projection beam to two well-chosen characteristic points of
projection image 70. One characteristic point might be the projected center of
computer screen image 50; the other one might be the projected version of a
point half-way the left vertical edge of computer screen image 50. Simple
application of the cosine rule (see Fig. 23, depicting a top-view of Fig. a
are known (the latter angle following from knowledge of the optical dimensions
of the projection beam), all other quantities in the triangle can be calculated
– including the sought-after angle b.
Given the above, it will be appreciated that inclusion of any such sensors will
allow a unique determination of position, shape, orientation and size of
projection image 70 (and a near-coincident interaction region 71 and interaction
structure 72), even in the general case of a projection device 40 (integrated in
the above described sense with at least base station 30) that is misaligned with
respect to projection region 60 both in the horizontal and vertical sense. Note
again that this may be done independent of whether or not the vertical and/or
horizontal keystone effect is corrected for.
[00181]
Finally,
then, it will be understood that use of an image capturing device 206 of which
the position and orientation in the x y z coordinate system is known may be of
great help in the presently discussed embodiment, as already alluded to in
paragraph 170 and 171. Indeed, those skilled in the art will recognize that
distance measuring device 206 may be embodied by combining knowledge of the
optical characteristics of projection device 40 (such as the dimensions of its
projection beam and the size of the to-be-projected image) with information
obtained from a captured image of projection image 70 by a sensor like image
capture device 206. After all, if the size of projection image 70 (for ease of
explanation, but not out of necessity, assumed to be rectangular) can be
established using an image capture device 206, one may use knowledge of the
optical characteristics of projection device 40 to easily establish the distance
from its optical origin 402 to, for example, the projected center of computer
screen image 50. This way, then, 3D parameters describing projection image 70,
interaction region 71 and interaction structure 72 can easily be determined, as
will be appreciated by those skilled in the art. Alternatively, focus mechanisms
in image capture device 206 may also be found useful in this regard. In short,
then, it will be understood that any device capable of establishing the distance
from a known point in the x y z coordinate system (such as, for example, the
optical origin 402 of the projection beam of projection device 40) to a
characteristic point of projection image 70, interaction region 71 or
interaction structure 72 is contemplated by the current application. It will
also be understood that, although the above continuously referred to the
projected center of computer screen image 50, this was merely intended as an
example and such a point may, without loss of functionality, be replaced by any
point related in a known fashion to computer screen image 50, projection image
70, interaction region 71 or interaction structure 72.
[00182]
Furthermore,
other techniques disclosed in this application may be combined with the above
described methods, including but not limited to those outlined in paragraphs
159, 165, 169, 170, 171, 172, 173 and 174. In particular, calibration sensors
associated with projection device 40 or base station 30 and their associated
calibration methods may be combined with methods related to a user-wielded
pointing device 20, as disclosed in this application. For example, the system
may require the user to highlight one or more calibration marks 721a,b,… with
a user-wielded pointing device 20, in order to correct for the horizontal
non-alignment of projection device 40 and projection region 60 (the horizontal
keystone-effect), while sensors incorporated in projection device 40 or base
station 30 may be used to automatically correct for any vertical misalignment.
Also, control points (see also paragraph 159) may be used to check the validity
of the calculated position, orientation, shape and size of projection image 70,
interaction region 71 and/or a near-coincident interaction structure 72. Also,
it will be understood that in certain embodiments base station 30 may be left
out entirely. Furthermore, distance measuring device 206 may also advantageously
be left out. To explain, the reader is referred to the embodiment described in
paragraphs starting at paragraph 120. If the methods and devices described there
are combined with the methods and devices outlined in paragraphs 177 – 182,
another advantageous embodiment will result. That is to say, if the position of
a user-wielded pointing device 20 is not too different from the optical origin
402 of the projection beam of projection device 40, such that they may be
considered to substantially coincide, it will be appreciated by those skilled in
the art that a system results that does not have a need for position sensing, an
associated base station 30 or distance measuring device 206. Particularly when
projection region 60 is sufficiently far away from projection device 40, such
that projection image 70 is relatively large, any small discrepancy between the
position of a user-wielded pointing device 20 and the position of projection
device 40 will be negligible. In such a case, knowledge of the dimensions of the
projection beam (assuming that the angle between its optical axis 401 and
projection region 60 is sufficiently close to 90 degrees) will be sufficient to
substantially ascertain the point-of-aim of user-wielded pointing device 20, as
will be obvious from the methods disclosed in this application. An improvement
on the above system may be obtained by including one or more distance sensing
devices 206 that may, for example, be used to establish the horizontal and/or
vertical angles between optical axis 401 and projection region 60, as described
above. This way, it is possible to compensate for a non-perpendicular alignment
that might, without inclusion of such sensors, cause intolerable errors in the
computation of the point-of-aim of a user-wielded pointing device 20.
